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ABSTRACT 



Critical thinking comprises the mental processes, 
strategies, and representations people use to solve problems, make 
r«mi?if2 S Xu an d lea f n ne * concepts. The study of critical thinking 

he educational, philosophical, and psychological traditions 
R :. Ennis °, ££ers a Philosophical taxonomy suggesting that 
critical thinking results from the interaction of a se* of 

2i?? ? r? 1 !!! S < .i 0ward « itical thinking with a set of abilities for 
SUfiS! ♦I 'S-??' * hll f R# st « rnb * r 9's psychological taxonomy 
defines the skills involved in critical thinking to be of three 
Kinds: metacomponents , performance components, and 
knowledge-acquisition components. Bloom's taxonomy of education puts 
knowledge at the lowest level, followed by comprehension, P 
application, analysis, and synthesis, with evaluation at the highest 
level. Tests for measuring critical thinking also come from the 
philosophical and psychological areas. The matson-Glaser Critical 
Thinking Appraisal, the Cornell Critical Thinking Test, and the New 

Ead?L2!"\£ f S eason i n 9 skills •« *«ived fro. the philosophicS 
tradition. The Triarchic Test of Intellectual Skills is 
psychologically derived, but does not try to separate critical 
?ni?„i! 9 r^? intelligence. Programs for training critical thinking 

i2Si«f r.f??i^ S -i2 78 T S OU f Se x ?, l0 9 ic ' Bradford and Stein's 1984 
course called "The ideal Problem Solver," Sternberg's 1986 

rSu erS i and J2L and Incre »si«9 Intelligence," and Nhimbey and 
Loehhead's 1982 "Problem Solving and Comprehension." Educators 
current concern with critical thinking offers students a new chance 
fnJ« de T el ° pin9 * Cri l 1Cal thioki «9 ^ills, but training must be brought 
i t ? iu aSp ? CtS o £ . the classroom to be successful. (Tables outlining 
E.J. Gubbins's Matrix of Thinking Skills, and Sternberg's program fo? 
training intellectual skills are appended.) (SRT) program xor 



********************* 
* 
* 



. — ** 

Reproductions supplied by EDRS are the best that can be made * 

from the original doc— ent . * 



ERIC 



Critical Thinking, Page 1 



CO Critical Thinking: Its Nature, Measurement, and Improvement 

rvi 



Robert J» Sternberg 
Tale Univers ity 



U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and imorovement 

EDUCATIONAL RESOURCES INFORMATION 
• CENTER (ERIC) 

^wThis document has been reproduced as 
(received from the person or organization 
originating it 

□ Minor changes have been made to improve 
reproduction quality 

• Points of view or opinions staled in this docu- 
ment do not necessarily represent official 
OERl position or policy 



Running head: Critical Thinking 



ri 

cr 
- 1 

ERIC 



Send proofs to Robert J. Sternberg 

Departaent of Psychology 
Tale Univers ity 
Box 11A Tale Station 
New Haven, CT 06520 



Critic*} Thinking, Page 2 



Critical Thinking: Its Nature, Measurement, and Improvement 

More than a decade has passed since then-President John F. Kennedy 
ordered the invasion of the Bay of Pigs. The invasion was to become one 
of the great disasters in U.S. political and military history. The 
invasion did not, of course, succeed in the ultimate overthrow of Cuba's 
Fidel Castro. More interestingly, there is a widespread consensus among 
students of the situation that the invasion never had a chance to succeed 
in the first place. The decision to invade, made largely by Ivy-Leagve 
educated men with aome experience in political affairs, represented what 
from almost sny point of view would have to be labeled as a lapse in 
critical thinking. 

What is critical thinking, and how can well-educated men and women 
show lapses in it that are serious enough to lead to fiascos such as the 
Bay of Pigs, the Watergate break-in and coverup, and any of a number of 
other such similar events in our country's history? The goal of this 
article is to define critical thinking, to review alternative approaches 
to understanding it, to compare some alternative procedures for measuring 
it, and to discuss gome alternative attempts to train it. 

A Definition of Critical Thinking 
Construed broadly, critical thinking comprises the mental processes. 
strategies., and rep resentations people use to solve problems, make 
decisions, and learn ne w concepts . The particular elements of critical 
thinking that people use vary widely both in scope and in quality across 
persons, tasks, and situations. Hence, it is necessary to specify in some 
detail just what the elements of critical thinking are, and how they vairy 
across persons, tasks, and situations. Such a specification is the goa]L 
of the next section of this article. 
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Theories of the Mature of Critical Thinking 
In tone fields of educational endeavor, it is difficult to get 
educational theorists to agree about anything. The field of critical 
thinking is distinctive for its amount of consensus among theoriests 
regarding the nature of critical thinking. This is not to say .hat the 
consensus is complete, or that alternative theories and approaches to 
theorizing are nonexistent. It is to say, however, that the agreements 
clearly outweigh the disagreements* A review of theories and approaches 
suggests that the major differences are in how broadly or narrowly the 
construct of critical thinking is viewed — in its boundaries — rather than 
in what is viewed to be the core. 
Three Traditions of Theorizing 

The study of critical thinking is of particular interest because of 
its confluence of three traditions of thought — the educational, the 
philosophical, and the psychological* Indeed, if there is a modern-day 
founder of the "critical-thinking movement,* 9 it is almost certainly John 
Dewey, who was simultaneously an educator, a philosopher, and a 
psychologist* 

The philosophical tradition * The concern of philosophers with the 
elements of critical thinking dates back to ancient times* If Dewey is 
the modern-day founder of the critical-thinking movement, then Plato and 
Aristotle would be its ancient founders* In more recent times, 
philosophers such as Ennis (in press), Lipman (in press), and Paul (in 
press) have devoted their attention to understanding the bases of critical 
thinking* 

Philosophers have focused their attention not so much upon the 
requirements of critical thinking in the classroom, but upon the 
requirements of formal logical systems* The difference in emphasis is 
ER s ^C"P ort * nt for two reasons* 4 



Critic. 1 Thinking, Page 4 



Fir.t. th. r.,.ir.„.t. of form.l logic.l . y .t,m. .„ not „ c .... rlly 
c.rre.po.d to the r .,„ ir „.. t . or .. MMIU1 .. „ f cb . Uren ta clis<roM 
•itu.tio... I.d„ 4 . the two ..t. of r.,uir...ot. .. y „. co . pIetely 
different. For e,.. p le, "ro.olotio. logic" provide. . po ..r,ul method for 
Proving certain logic.l theorem.. but pfob . bl , .„ „. (u ^ ^ 
mind.) would ...i. tb . t cbildren „. r «. olot . oa Iotic> ^ 

even th.t „.., of the. would ,poot.n.ou. ly edopt it .ft.r envthing but 
exten.ive tr.ini.g. Hot .U phiio.oph.r. h.v. been , uiclt to ^ 
difference between the 1... of logic t „. „„ „ f thought 

»ool. (1,54, „ titl . a hi . book on „ 8oolein iog . c _„ Jj!ij!a2 ^ tJU!£smi 
de.pit. the f.ct th.t there i. no evidence .t .11 th.t people 
• pont.n.oo.ly .dopt th... l.v. in tbe£r tbought . 

Second, the retirement, of logic.l , y . t ... m.y p.rh.p. h.tter be 
tbonght of .. providing ..del. of c sx££l£s££ r . ther tb .„ #f 
ZZlisr^ for ta... thonght. The role, of l.gi. ... tell „ „„ people 
might thinh critic.ll, nnd.r ide.l circum.t.no in which th. li.it.tl.n. 
tvpie.ll, pieced npon th. „.... inf.rm.tio. proc.in, „. Mt in 

Pl.ce. ».t there .re num.rou. potenti.l Umit.tion. th.t ordin.ril, Hock 
the otilix.tion of onr foil co.p.t.nc.-limit.d time, li.ited i.,.r..ti.n, 
limited working memory c.p.citv, limited motiv.tion, .nd to on. 

Th... two d.li.it.tion. on th. interpr.t.tion of philo.ophic.l 
th.orie. .re not critici... of philo.ophic! .ppro.ch... w. need to know 
the m.ximnm potenti.l. of critici thooght, le.t .. ..ttle for 1... 
pr.ci.ion .nd reflectivity in our thinking th.n th.t of which .. .re 
c.p.ble. At th. .... time. ., .... to r.cogni.e th. p.r.on.l ,nd 
•itu.tion.l con.tr.int. tb.t often i.pi.g, upo . our vorki „ g op to fuU 
capacity. 
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The psy chological tradition , Psychology* ta interested in the nature 
of critic?* thinking, such as Bransford (1984), Bruuer (I960, 1961 > 9 
Feueratein (1980), and Sternberg (1985), have been particularly concerned 
with characterizing critical thinking as it is performed under the 
limitations of the person and the environment. For example, Feuerstein 
(1980) has specified how the critical thinking of retarded performers 
differs from that of normal performers; Sternberg and Daridson (1983), in 
contrast, compared the cr it ical thinking of gifted and normal performers * 
None of these theorists, though, has proposed a model of totally rational 
thinking. Indeed, Guyote and Sternberg's (1981) work is more typical of 
psychological theorising in pinpointing how people differ from the fully 
rational performer in solving syllogisms* 

Psychological theorising can be valuable in showing how people think 
critically in the absence of full information, unlimited time, perfect 
memory, and so on* At the same time, it is necessary to observe tvo 
cautions in evaluating the theories of many psychologists* 

First, the theories of psychologists are often derived from and tested 
on performance of human subjects in laboratory settings, and there is no 
guarantee that people will perform in their everyday lives and especially 
in the classrocm in the same ways that they do in the laboratory* To the 
contrary , most available evidence suggests consequent ial differences in 
the two kinds of settings of performance* 

Second, the constraints of proposing theories that are empirically 
testable through the standard means of psychological experimentation 
sometimes results in theories that oversimplify the analysis of critical 
thinking* The constraint of testability contributes to scientific 
analysis but often at the expense of oversimplification* 
O 
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The educational tradition. In the educational tradition of theorising 
are leading figures auch as Bloom (1956), Gagne (1965), Perkins (1981), 
and Rensulli (1976), whose theorising seems directly responsive to the 
skills needed by children in the classroom for problem solving, decision 
making, and concept learning. Bloom's (1956) famous taxonomy of cognitive 
skills snd Gagne'a (1965) well-known hierarchy of learning skills have 
seen widespread application in classroom situstions and even textbook 
creation. These theorists have drawn heavily upon classroom observation, 
text analysis, snd process analysis of thinking in the clsssroom to guide 
their thinking shout critical thinking. 

Educational theories have the advantage of being closely tied to 
classroom observation and experience. At the same time, there are two 
points to keep in mind when using or evaluating these theories. 

First, the educational theories often do not have the clarity in 
epistemological status that is chsracteris tic of the philosophical and 
psychological theories, making it more difficult, in some respects, both 
to evaluate and to use the educational theories. Philosophical theories 
tend to be competence theories specifying what people can do; 
psychological theories tend to be performance theories specifying what 
people actually do; educational theories are often a mixture of the two, 
with the nature and proportions of the mix less than clearly specified. 
To this day, for example, educators argue over the extent to which Bloom's 
taxonomy represents a prescriptive versus a descriptive model of human 
thought . 

Second, in my experience, educationally-based theories tend not to 
have been subjected to tests of the same degree of rigor that has 
characterized the testing of philosophical and paychological theories. 
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Philosophical theories based on various kinds of logics oust be logically 
rigorous and internally consistent* Psychological theories based on human 
performance must be externally consistent with respect to the behavior 
they purport to describe* Educational theories are often not subjected 
either to the logical tests of philosophical theories or to the 
psychological tests of the psychological theories* 
A Framework for Theories of the Nature of Critical Thinking 

Because there are so many accounts of critical thinking, and because 
they so often say similar things in different ways, or even occasionally 
different things in similar ways, it becomes important to develop some 
kind of framework that can encompass the various theories, and highlight 
their similarities and differences. 

The framework proposed here is based upon that generated by my 
triarchic theory of human intelligence (Sternberg, 1985). The framework 
was derived in large part by classifying the goals and scope of many 
previous theories of intellectual functioning* In the present context, it 
is proposed that theories of critical thinking can, and often do deal with 
one or more aspects of critical thinking — its relation to the mind of the 
individual, its relation to the context in which it occurs, and its 
relation to the experience of the individual with various kinds of tasks 
and situations previously confronted that required critical thinking in 
greater or lesser degree* 

The relation of critical thinking to the internal world of the 
individual * Theories of the internal workings of the mind when it engages 
in critical thinking can be seen as being aimed at the very essence of 
what critical thinking is about: What do we do when we think critically, 
and how do we do it? g 

O 
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Student, of critical thinking have proposed various taxonomies of 
•kill, purported to .pan the range of critic.l thinking. Con.ider three 
eaa.ple. of .uch taxonomies, one each fro* the philosophical, 
p.ychological, and educational tradition.. 

A philosophical taxonomy: R ob »r, gmj. 1 T~^ Enni., a 

Philo.opher, ha. .ugge.ted that critic.l thinking re.ult. fro. the 
inter.ction of a .et of di.po.itions toward critic.l thinking with a set 
of abilitie. for critic.l thinking. 

The di.po.ition. include. a.ong other., (.) .eeking a clear statement 
of the the.i. or que.tion, (h) .eeking re..on*, (c) trying to he 
well-informed, and (d) trying to rem.in relevant to the ..in point. The 
idea underlying the li.ting of di.po.ition. is that . prerequisite for 
critic.l thinking is the .otivation or desire to think critically. 

Ennis classifies anilities under five main categories, which are 
themselves further subdivided. The categories are elementary 
clarification, basic support, inference, advanced clarification, and 
strategy and tactics. 

Elementary clarification consist, of focusing on a que.tion, analyring 
•rgu.ent., .nd ..king and an.wering questions of ci.rif ic.tion and/or 
challenge. Ba.ic . upp ort involves judging the credibility of a source .nd 
ob.erving and judging ob.erv.tion report.. Inference compri.es deducing 
•nd judging deductions, inducing and judging inductions, and ..king and 
judging value judg.ents. Advanced clarification involves defining terms 
•nd judging definitions, and identifying assumptions. Finally, atr.tegy 
•nd tactic, include deciding on an action and interacting with other.. 
The.e c.tegorie. are .11 them.elve. further .ubdivided (.ee Enni., in 
press) . 
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2 • A psychological ta xonomy: Robert Sternberg (1985) . My ovn 
taxonomy derives not from « logical but from a paychological analysis of 
critical thinking. According to ay ovn "componential " account of thought, 
the skills involved in critical thinking are of three kinds: 
aetacoaponents, perforaance coaponents, and knowledge-acquisition 
component s . 

Metecoaponents are higher order executive processes used to plan what 
one is going to do, aonitor it while one is doing it, and evaluate it 
after it is done. The aetacoaponents include recognising that a problea 
exists, defining the nature of the problea, deciding on a set of steps for 
solving the problea, ordering these steps into a coherent strategy, 
deciding upon a fora of aental representation for inforaation, allocating 
one's tiae and resources in solving a problea, aonitoring one's solution 
to a problea as the problea is being solved, and utilising feedback 
regarding problea solving after one's problea solving has been coapleted. 
Siailar taxonoaies have been proposed by Brown (1978) and Bransford 
(1984). 

Perforaance coaponents are lower order, nonexecutive processes used to 
execute the instructions of the aetacoaponents, and provide feedback to 
thea. Perforaance coaponents vary by domain of perforaance, for example, 
inductive reasoning, deductive reasoning, spatial visualisation, reading, 
and so on. Consider, for exaaple, the perforaance coaponents of 
induction. These include encoding stiauli, coaparing stiauli, inferring 
relations between stiauli, mapping relations betweea relations, applying 
relations froa one doaain to another, justifying potential repsonses, and 
responding. 



10 
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Knowledge-acquisition components are the processes used to learn 
concepts or procedures. Three such components are selective encoding, 
which involves screening relevsnt from irrelevant information, selective 
combination, which involves putting together the relevant information in a 
coherent and organised way; and selective comparison, which involves 
relating old, previously known information to new, about to be learned 
information. 

3 * An ^"cations; taxpnpm Y : Bloom (1956) . Bloom has proposed a 

hierarchical taxonomy for cognitive information processing. At the lowest 
level is knowledge. The n *xt level if comprehension, which requires one 
to go beyond knowledge in that one must understand what one comes to know. 
At the next level is application, which is a level higher yet in that the 
individual must also be able to apply what he or she has comprehended. A 
level higher up is analysis, which requires one critically to appraise 
what one comprehends and applies. Still higher is synthesis, which 
requires putting together in a somewhat creative way the knowledge one has 
analysed in various domains. At the highest level is evaluation, which is 
a broad and critical appraisal of the knowledge one has analyzed and 
synthesized. 

In this section, I have described three taxonomies of critical 
thinking skilla. Although the organizations of these taxonomies are 
different, as sre the exact thinking skills they comprise, the overlap 
among the taxonomies is striking. All of the theorists cited believe in 
the importance of learning, comprehension, deductive reasoning; and 
inductive reasoning skills. The names they give to the various skills 
within esch of these domsins differ, but the skills seem to differ hardly 
at all, except with respect to how finely dif fereatiated and how broadly 

O 1 1 
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encompassing they are within one or another theory. Thus, it appears 
that there nay be a certain core of critical thinking skills that would 
appear in any reasonably complete list. In fact, Joan Gubbins of the 
Connecticut State Department of Education has surveyed a large number of 
similar taxonomies, and has developed a list that reflects the skills 
listed by numerous theorists. Table 1 contains the various thinking 
skills in Gubbins's (1985) list. 



Insert Table 1 about here 



The relation of c ritical thinking to the experience of the 
individual. It is one Lfcing to apply critical thinking to tasks and 
situations that are familiar to us, but quite another to apply it to tasks 
and situations that are unfamiliar. Often, the processes and strategies 
that come so easily to ua in familiar situations simply resist 
implementation in strange tasks and situations. The processes and 
strategies may or may not be different in the novel task or situation. 
Sometimes, it is their readiness for implementation that differs between 
familiar and unfamiliar situations* Again, consider three alternative but 
related views of the role of novel experience in critical thinking. 

!• A phil osophical view: Paul (in press) . According to the 
philosopher, Richard Paul, an essertial element of critical 
thinking — perhaps the essential element—is the ability to see things from 
others' points of view, which may be quiet novel and even foreign with 
respect to one's own point of view- Paul refers to cuch thinking as 
dialottical. For example, the ability of a liberal to see things from a 
conservative point of view, the ability of a husband to take his wife's 
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point of viev on the desirability of sharing housework, and the ability of 
•n adult to see things through a child's eyes, are all example, of 
di.logic.l thinking. Such thinking is necessary to escape the egocentri.m 
«nd narrovness of perspective that characterises the "uni logica 1 " thinking 
in which most of us so often indulge. 

2. A_j>a^choio Kical yiev: SternberR (1985).. I n my ova> experiential 
account of critical thinking, the importance of coping vith novelty is 
strongly emphasised. There is good reason for this emphasis: It is often 
in the novel task or situation-such as the Bay of Pigs or the Watergate 
crisis mentioned earlier— that the potentials for either greai gain or 
serious loss present themselves. Indeed, the great contributions to the 
world are often traceable to major insights in which an ingenious 
individual has seen a new and useful way to solving a problem, whether a 
new one or an old one. 

According to Sternberg and Davidson (1983) (see also Davidson & 
Sternberg. 1984). insight is an important part of the ability to deal with 
novelty. We propose that insights are of three kinds-selective encoding, 
selective combination, and selective comparison, as discussed earlier. 
Thus, the processes of insight do not differ from the process of knowledge 
scquisition in more ordinary critical thinking. What does differ, 
however, is the knowledge base to which these processes can be brought to 
bear. In ordinary thinking, we have a set of clues or guidelines or rules 
we can use to help us learn new things. In insightful thinking, we do not 
have such a readily-available set of clues, guidelines, or rules. We must 
mske up the rules a. we go along, and it is in this sense that an insight 
is a leap into the unknown: It is a leap not only in knowledge, but in 
the way in which that knowledge is acquired. 

* 13 
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3* An educational view: DeBono (1967. 1969) . DeBono has proposed a 
aeriea of techniques for enhancing people's critical thinking. One of the 
most well-known of these is what he refer? to as PMI: plus, minus, 
interesting. What, exactly, does this mean? 

Consider a series of alternative solutions to a given problem, for 
example, what we would do if all money in the world instantly became 
worthless. (DeBono's interest in developing the ability of his readers 
and listeners to deal with novelty can be seen right away in his choice of 
problems, many of which are themselves highly novel, as is this one.) 
DeBono suggests that as each alternative solution to a complex problem is 
posed, one should list the positive (glus), negative (minus), and 
.interesting features of that solution. DeBono's view is unusual for its 
stress on the evaluation of the interest as well as the positive and 
negative features of each solution. Getting people to think in this way 
encourages them to develop their ability to see both familiar and 
unfamiliar problems in novel and potentially interesting ways. 

Once again, underlying the differences in language, there appears to 
be a common core of beliefs regarding the nature of the ability to deal 
with novelty. Insights almost inevitably involve some degree of 
dialogical thinking. They require us to perceive a problem we may have 
been pondering for quite some time from a new and different perspective. 
Most of the classical insight problems, such as the "nine-dot" problem, 
require us to see a given problem from a new and seemingly unusual vantage 
point. In the nine-dot problem, for example, one must connect nine dots 
arranged in a 3 x 3 matrix with no more than four pencil lines, without 
ever lifting one's pencil from the paper. Solution of the problem 
requires one to recognise that one must go outside the implicit perimeter 
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defined by the dots in order to be able to connect then. Similarly, truly 
insightful thought* are generally interesting thoughts, even if the 
insights turn out to be wrong. Indeed, encouraging people to think in 
"interesting" ways may be tantamount to encouraging them to think more 
insightfully. Thus, whatever the language the various theorists use, 
their conception of the role of novelty in critical thinking seems to be 
highly similar, although certainly not identical. 

The relation of critical thinking to the external world of the 
individual . Perhaps the single question that most directly motivates this 
section of the article is: critical thinking for what? If innumerable 
studies of transfer of training have revealed anything, it is that 
transfer is exceedingly difficult to attain. Teaching thinking skills or 
any other skills in any one context does not assure or even render likely, 
their transfer to another context. Moreover, it is not even immediately 
obvious that critical-thinking skills are the same in all situations. 
Certainly, their instantiations differ. It is necessary that critical 
thinking skills be taught in a way that maximises the probability of their 
transfer to real-life situations. This necessity has been recognised in 
different ways by different theorists. 

*• A philosophical approach : Lioman (1974) . Lipman's program for 
training thinking skills, Philosophy for Children, presents these skills 
in the context of the everyday lives of children (see Lipman & Sharp, 
1975; Lipman, Sharp, & Oscanyan, 1977). The basic format of the program 
has students resding novels about children in their everyday worlds, and 
about how they bring critical thinking to bear upon these everyday 
worlds. Students are not left to figure out how to bridge the gap between 
critical thinking skills and everyday life. Rather, in all phases of the 
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program, they are provided with explicit models (the protagonists in the 
novels, who are themselves children) of how this bridging can be done. 
2 • Two psychologica l approaches; Bransford (1984) aad Sternberg 

In Bransford's book on the "ideal" problem solver, almost all of 
the techniques presented are demonstrated through everyday examples. The 
techniques for problem solving are "brought to life" through concrete 
instances, and readers are encouraged immediately to apply the techniques 
to problems they face in their own lives. 

Sternberg (1985) takes a somewhat different approach, initially 
illustrating methods of critical thinking through concrete examples, but 
then providing exercises that range from the academic to the practical. 
The idea motivating this variation in types of context is that students 
will best learn how to apply processes and strategies of critical thinking 
in their everyday lives if they use these processes and strategies in the 
broadest possible array of circumstances, ranging from the most academic 
to the most practical. 

Both Bransford's and Sternberg's approaches contrast with the approach 
of Feuerstein (1980), in which problems (or "instruments," as Feuerstein 
calls them) are largely academic and abstract in nature. Feuerstein calls 
for bridging of cognitive skills to students' everyday lives, but this 
bridging is pretty much left to the teacher, and thus is less controlled 
in terms of how and how much it occurs. From the present point of view, 
this deemphasis upon the external world of the individual in the training 
would tend to have less favorable implications for transfer than would the 
greater emphasis of other programs. 
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3 * An educational approach: Head Start . Perhaps the best example of 

educational philosophy for bridging the gap between thinking skills and 
the real world was embodied in the numerous Head Start programs of the 
1960s, tome of which continue in modified form to the present day. There 
was no one consistent philosophy or even psychological theory behind these 
programa. Many programs seem to have had no particular ph iloaoph ica 1 or 
theoretical underpinnings at all. What the programs did have was a 
commitment on the part of their initiators to making a difference in 
children's lives, and particularly, in their schooling. However one 
evaluates the outcomes of these programs, one would have to give them 
credit for bringing educators and laypeople to the realization that 
intellectual skills are potentially trainable. These programs were 
tranaitional between nothing, on the one hand, and the theory-based 
programs of today, on the other. 

Tests for Measuring Critical Thinking 
Several tests have been advanced that purport to measure 
critical-thinking skills. The tests overlap to a large degree in the 
akilla they measure. Nevertheless, there are some differences worthy of 
note. The purpose of this section is not to provide an exhaustive review 
of all of the available tests, but rather to illustrate some of the basic 
principles underlying these tests, and how they relate across the range of 
tests available. The emphasis will be upon tests at the secondary-school 
level . 

1 . Three philosophical lv-d erived teata; the Wataon-Claaer Critical 
Thinking Appraisal, the Cornell Cr itical Thinking Test, and the Hew Jersey 
Test of Reasoning Skills. All three of these tests are derived from the 
philosophical tradition of measuring critical thinking* They are highly 
overlapping conceptually. 
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The Vatson-Glaser Critical Thinking Appraisal (Watson & Glaser, 1980) 
contains five subtests, each measuring a somewhat different 
critical-thinking skill. There are two forms of the test, each containing 
a total of 80 items. Untimed administration is recommended, although an 
optimal time limit of 40 minutes can be imposed (in vhich case subtest 
scores will be less in terpretable , as timing is for the test as a whole 
and subjects may not complete later subtests). The test is suitable for 
individuals at the Grade 9 level and above. 

The five subtests are (a) Inference, which requires discrimination 
among degrees of validity of inferences drawn from given data; (b) 
Recognition of Assumptions, which requires recognition of assertions; (c) 
Deduction, which requires determination of given statements of premises; 

(d) Interpretation, which involves weighing of evidence and deciding if 
generalisations or conclusions based on the given data are warranted; and 

(e) Evaluation of Arguments, which requires distinguishing between 
arguments that are strong and relevant and those that are weak or 
irrelevant to a particular question at issue. The actual test items have 
high face validity, in that they draw upon classroom and general life 
situations • 

Various norms are available, including high school norms (by grade), 
norms for various college and university groups, and various occupational 
norms. The main kind of score is the percentile equivalent to a giver raw 
score. Split-half reliability coefficients for various groups generally 
range in the ,70's for the test or a whole. Correlations with 
intelligence tests are variable, but seem to center at about the .6 level 
with verbally-weighted tests. It is not clear whether the test has 
incremental validity in predicting various kinds of performances beyond 
that which would be obtained with a student group intelligence test. 

o 18 
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The Cornell Critical Thinking Teat (Enni. 4 Millm.n. 1971) i. baaed 
upon Enni.'. conception of critical thinking, as briefly de.cribed 
earlier. The test i. available in two levels. X and Z. Level X is 
appropriate for secondary school (grade 7) and beyond. Level Z primarily 
for college students (and bright secondary-school students). Level X ha. 
71 questions and a time limit of 50 minutes. Level Z has 52 questions, 
but the same time limit as Level X. 

Level X contain, four sections. The first section contains items 
asking for the bearing, if any. of information on an hypothesis. The 
hypothesis is in every case a general statement. Examinee, must indicate 
whether a particular hypothe.i. i. warranted by the data. The .econd 
section is concerned with mea.uring examinee.' ability to judge the 
reliability of information on the ba.i. of it. .ource and the conditions 
under which it i. obtained. The third .ection mea.ure. students' ability 
to judge whether a statement follows from its premises, and the fourth 
section involves identification of assumptions. 

Level Z contains seven sections, measuring the examinee.', ability to 
(a) indicate whether a .t.tement follow, from it. premi.e.. (b) detect 
equivocal argument., (c) evaluate reliability of ob.ervation. and 
authenticity of .ource.. (d) judge the direction of .upport. if any. for a 
given hypothe.i.. (e) focu. on choo.ing of u.eful prediction, for 
hypothe.i. te.ting, (f) define term., and (g) spot gaps in argument.. 

Horm. for both level, of the Cornell Critical Thinking Te.t are give* 
-8 percentile equivalent.. Internal con.i.tency rel iabil it ie. for v.riou. 
group, appear to center around .8 for Level X. and around .7 for Level Z. 
Correlation, with other te.t. are variable. They em to center around .5 
for verbally oriented intelligence te.t.. The .epo ed correlation with 
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the Vatson-Glaser, .48, is no higher thsn the correlation of the test with 
verbal IQ and scholastic aptitude measures, , n d the correlation of the 
Cornell with the ACE Test of Critical Thinking, .44, is also no better 
than the correlation of the Cornell with the Watson-Glaser . These data 
are not auspicious in indicating a clear, dif f erent iable construct of 
"critical thinking" spart from general verbal intelligence. 

The third of the philosophically-based tests to be described here is 
the New Jersey Test of Reasoning Skills, Form B, developed by Virginia 
Shipman (1983) of the Educational Testing Service and promoted in 
conjunction with Lipman's Philosophy for Children program. The New Jersey 
Test of Reasoning Skills is a 50-item inventory purporting to measure 22 
different skill areas: converting statements, translating into logical 
form, inclusion/exclusion, recognizing improper questions, avoiding 
jumping to conclusions, analogical reasoning, detecting underlying 
assumptions, eliminating alternatives, inductive reasoning, reasoning with 
relationships, detecting ambiguities, discerning causal relationships, 
identifying good ressons, recognizing symmetrical relationships, 
categorical syllogistic reasoning, distinguishing differences of kind and 
degree, recognizing transitive relationships, recognizing dubious 
authority, reasoning with 4-possibilities matrices, contradicting 
statements, whole-part and part-whole reasoning, and conditional 
syllogistic reasoning. Like the other tests, this one is highly verbal. 
Its reliabilities are reported to be in the mid-to-high- .8<Ps, and it is 
reported to correlate at the .6 to .8 level with subtests of the New 
Jersey College Basic Skills Placement Test, which is a test verbal and 
mathematical skills emphasizing achievement at least as much as aptitude. 
The fact that the New Jersey Test of Reasoning Skills correlates at the .8 
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level with the "Reading Comprehension" and "Sentence Sense" s-btests of 
the placement teat aight be seen by tone as slightly disturbing: All 
three of the test, described so far are highly verbally loaded, and one 
might well wonder a. to the extent that *hat they measure is separable 
from general verbal .kills. Indeed, the little evidence accumulated ao 
far does not indicate a clear separation at .11, perhaps because a fairly 
high level of verbal comprehension is prerequisite for high scores on all 
of these tests. 

2 * A P«Ychologicanv-derivgd test; The Triarchic Test of 
Intellectual Skills . This test is new, is currently available in two 
forms only from the author (Sternberg), and ia not yet normed. The 
triarchic test is based upon Sternberg's (1985) triarchic theory of 
intelligence, and hence does not purport to separate critical thinking 
from intelligence. The test is appropriate for high school and college 
levels. The twelve untimed subtests of the triarchic test are equally 
divided between verbal and nonverbal content, and measure (a) 
metacomponential thinking skills (planning, monitoring, evaluating), (b) 
performance-componential skills (inferring relations, applying relations, 
mapping higher order relations between domaias), (c) knowledge acquisition 
componential skills (learning concepts in natural contexts), (d) ability 
to deal with novelty (diatinguishing relevant from irrelevant information, 
combining relevant information in a logical way, bringing 
previously-acquired knowledge to bear upon the acquisitioa and 
understanding of new knowledge), (e) automatization of information 
processing (making conscious and controlled processing, subconscious and 
automatised), and (f) adaptive flexibility (bringing the various kinds of 
akills described above to bear upon everyday adaptation, as in 
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route-planning and evaluating inferential fallacies in everyday 
reasoning). No normative, reliability, or validity information IV e ye 
available . 

To conclude, several tests are available for measuring critical 
thinking skills. The philosophically-based ones are highly verbally 
loaded, but measure reasoning in the verbal context rather than straight 
knowledge or fact comprehension. The dis tinguishabil ity of their scores 
from verbal intelligence is marginal. The psychologically-based test 
contains both verbal and nonverbal test items. No attempt is made in this 
test to distinguish between critical-thinking and intellectual skills. 
All of the tests provide means for assessing reasoning without heavy 
demands upon students' knowledge base. 

Programs for Training Critical Thinking 
Programs for training critical thinking skills have been with us for 
thousands of years, although they have not always been recognised as 
such. The traditional name for such programs has been "logic," and at the 
college level, such courses have usually been taught in philosophy 
departments. A complete review of programs for training cr it ical- thinking 
skills would obviously be beyond the bounds of this article (but see 
Nickerson, Perkins, & Smith, in press; Wagner & Sternberg, 1984). 
Nevertheless, it is possible to say something about the range of such 
programs. The emphasis here will again be at the secondary level. 

*• A philosophicall y-based progra m ! Cooi (1978) . At the 
secondary-college level, courses in logic have traditionally served the 
function of developing students' critical-thinking skills. Texts on logic 
remain among the best of the philosophically-based programs for secondary 
and college students. One of the most well-known such texts is Copi 
(1978), which has gone through five editions. 
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people typically encounter. Moreover, logic courses fail fully to take 
into account the performance limitations on human competence, (e.g., 
memory capacity or time or priorities for problem solving), 
Paychologically-based courses on critical thinking try to go beyond 
straightforward logic courses to deal with some of these problems. 

2 - Paychologicallv-based pro grams; Bransford and Stgin (1984) and 
Sternberg (1986). Many programs based upon psychological theories or 
principles have been proposed. Two of the most recent such courses are 
those of Bransford and Stein (1984) asd of Sternberg (1986). 

Bransford and Stein's course is called The Ideal Problem Solver , where 
IDEAL is an acronym for five steps in what Bransford and Stein seem to 
perceive as ideal problem solving: identifying the problem, Defining and 
representing the problem, Exploring possible strategies, Acting on the 
strategies, Rooking back and evaluating the effects of one's activities. 

The course is presented in a brief (150-page) paperback book 
containing 8 chapters, plus appendices, answers to exercises, and 
indices. The chapters cover (1) the importance of problem solving, (2) 
the model for improving problem solving, (3) improving memory skills, (4) 
learning with understanding, (5) intelligent criticism, (6) creat ivity , ( 7 ) 
effective communication, and (8) concluding remarks. The program is 
impressive for its lucidity, breadth, brevity, effective use of concrete 
examples, and connection to psychological theory and research. If the 
course has a weakness, it is perhaps in the limited coverage possible in a 
very brief text. This is a weakness only if one is seeking an in-depth 
course in critical thinking rather than a concise guide, which the IDEAL 
book certainly provide*** 
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Sternberg'. (1986) Pnder.t.nd in, ^ Ib^jjjL^ I.»n{ f r .,. dravs 
even .ore heavily upon psychological theory than doe. the IDEAL program. 
In the ca.e of Sternberg', program, the theory i. hi. own triarchic theory 
of human intelligence (Sternberg, 1985). The org.niration of the book and 
teacher.' guide are .hown in Table 2. Thi. program i. larger in .cope 
than i. Br.n.ford and Stein'.. It i. intended to .erve a. . year-long, or 
minimally, .eme.ter-long cour.e. Moreover, vherea. the IDEAL cour.e is 
de.igned prim.rily for individual reading by individual people, the 
Sternberg cour.e i. de.igned .uch that reading of the text (which can 
•tand on it. own) i. ideally .upplemented by cla.. di.cu..ion, paper., 
supplementary activitiea, and the like. 



Insert Table 2 about here 



Sternberg', program i. based upon .everal key in.truct ional 
principle. . 

First, one must teach for transfer, rather than merely hoping it will 
occur. The program doe. .o by including problem, that range from academic 
to practical, that range widely in content (e.g., ..thematic, logic, 
reading, .cience, .ocial .tudie.), that range fro. familiar to unfa.iliar, 
and that range fro. ab.tract to quite concrete. The idea ia to 
in.tantiate the basic processes of the triarchic theory in a. broad a 
range of proble. types as possible. 

Second, the program emphasises motivsting both students and teachers. 
A key motivational device is tesching the students and teachers about the 
theory and how it serves as a useful basis for a program to train 
intellectual skills. 
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Third, the program emphasizes training of me t acomponen tc (executive 
proceaaea) aa veil as performance components and knowledge-acquisition 
components (non-executive proceaaea) , aa well aa their application to 
novel and real-world situations. Thus, the inatruction covers the full 
range of stipuJations of the underlying psychological theory. 

Finally, the program has an entire chapter on emotional aad 
motivational blocka to the utilisation of one's intelligence. These 
blocks include diapoaitiona auch aa fear of failure, lack of 
follow-through, taak completion dif f icultiea, miaa ttribut ion of blame for 
failure, and the like. The goal of thia chapter ia to make students aware 
of the impediments that often prevent them from making the most of 
whatever intellectual skills they have. 

3 . An educat ionallv-baaed program: Whimbev & Lochhead (1982) . 
Problem Solving and Comprehension (third edition), a course at the high 
school-college level prepared by Arthur Whimbey and Jack Lochhead, is 
fairly typical of educationally-baaed programs for training 
critical-thinking akilla. It can be used as a main text or as a 
supplementary text on courses on critical thinking. The book contains 11 
chaptera plua appendices: (1) test your mind, (2) errors in reasoning, 
(3) problem-solving methods, (4) verbal reasoning problems, (5) six myths 
about reading, (6) analogies, (7) writing relationship sentences, (8) how 
to form analogiea, (9) analysis of trends and patterns, (10) solving 
mathematical word problems, and (11) the "post-wasi test." There is no 
particular paychological theory underlying the program, and the order of 
the chaptera a earns to be aomewhat arbitrary. Problems tend to be fairly 
academic, and no specific provisions appear to be built into the program 
to encourage transfer. Primarily, what students get is a lot of practice 
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in problem solving, and fairly minimal instruction in general techniques 
of problem solving. The course does not contain the fairly elaborate 
motivational aids of the two courses discussed above, nor does it contain 
many real-world problems. 

Conclusions 

I have presented in this article a brief overview of some of the main 
issues in the study of critical thinking: its nature, its measurement, 
and its improvement. Although there is still a long way to go, we have 
come remarkably far during the last few years in advancing our 
understanding of critical thinking. We have some good ideas both about 
how to test it and how to train it. At the same time, we need to 
recognize some of the limitations on our present understanding. 

First, we have a much better understanding of analytical (critical) 
thiuking than we do of synthetic (creative) thinking. This imbaltsce in 
our understanding is not for a lack of attempts to understand creative 
thinking (see e.g., Amabile, 1983; Perkins, 1981). Rather, creative 
thinking seems to be much more resistant to analysis. Yet, the most 
important contributions of thinking to the world and its cultures are 
probably in the synthetic domain rather than in the analytic one. 

Second, existing tests seem only to scratch the surface of critical 
thinking, and to do that even in flaved ways. We have seen how many 
existing tests tend to be highly "verbally loaded," and indeed, what these 
tests measure is not clearly distinguishable from verbal intelligence as 
it is traditionally operat ional ized in standard tests of intelligence. 
Moreover, there is a large gap between the ability to apply critical 
thinking in fairly trivial, highly structured, and usually multiple-choice 
tests, on the one hand, and in one's everyday life, on the other. Rone of 
the tests came even close to bridging this gap. 
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Third, our training programs for improving critical thinking arc 
themselves in need of improvement (aee Sternberg, in press, for a critique 
of these programs). The programs, like the tests, do not fully bridge the 
gap between the classroom situation and situations outside it. Moreover, 
vith few exceptions, the programs tend to be fairly narrow both in the 
range of skills they cover and in the instantiations within which these 
skills are instantiated. 

The current concern of educators with critical thinking offers 
students a new chance for developing critical-thinking skills. This 
chance will come to nought, however, if the concern proves to be nothing 
more than a brief infauation, if training in critical thinking is not 
brought into all aspects of classroom endeavor, or if *he concern stays 
only a concern and is not followed through with large-scale 
interventions. Training in critical-thinking should not be the privilege 
of a selected intellectual minority, or the luxury of the upper-class. It 
should be the right of every student, and it is our responsibility to all 
our students to enable them to exercise this right. 
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Table 1 

Gubbins's (1985) Matrix of Thinking Skills- 
Problem Solving 

A. Identifying general problem 

B. Clarifying problem 

C Formulating hypothesis 

D. Formulating appropriate questions 

E. Generating related ideas 

F. Formulating alternative solutions 

G. Choosing best solution 

H. Applying the solution 

I. Monitoring acceptance oc the solution 
J. Drawing conclusions 

II. Decision Making 

A. Stating desired goal/condition 

B. Stating obstacles to goal /cond it ion 
C Identifying alternatives 

D. Examining alternatives 

E. Ranking alternatives 

F. Choosing best alternative 

G. Evaluating actions 
III. Inferences 

A. Inductive thinking skills 

1. Determining cause and effect 

2. Analysing open-ended problems 

3. Reasoning by Analogy 
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4. Making inferences 

5. Determining relevant information 

6. Recognizing relationships 

7. Solving insight problems 
B. Deductive thinking skills 

1. Using logic 

2. Spotting contradictory ststements 

3. Analyzing syllogisms 

4. Solving spatial problems 
IV. Divergent Thinking Skills 

A. Listing sttribates of objects/situation 

B. Generating multiple ideas (fluency) 

C. Generating different ideas (flexibility) 

D. Genersting unique idea* (originality) 

E. Generating detailed ideas (elaboration) 

F. Synthesizing information 
V. Evaluative Thinking Skills 

A. Distinguishing between fscts and opinions 

B. Judging credibility of a source 

C. Observing and judging observstion reports 

D. Identifying centrsl issues and problems 

E. Recognizing underlying assumptions 

F. Detecting biss, stereotypes, cliches 

G. Recognizing loaded language 

H. Evaluating hypotheses 

I. Classifying data 

J. Predicting consequences 

34 



ERIC 



Critical Thinking, Page 34 



K. Demonstrating sequential synthesis of information 
L. Planning alternative strategies 

M. Recognizing inconsistencies in information 
N. Identifying stated and unstated reasons 

0. Comparing similsrities and differences 

P. Evaluating arguments 
VI. Philosophy and Reasoning 

A. Using dialogical/dialectical approaches 

"This matrix is based upon a compilation and distillation of deas from 
Bloom. Br.nsford. Bruner. Carpenter. Dewey. Ennis. Feuerstein. Jones. 
Kurfman. Kurfman & Solomon, Lipmsn, Orlandi, Parnes, Paul, Perkins, 
Renzulli, Sternberg. Suchman. Taba . Torrance. Upton, the Ross Test, the 
Whimbey Anelytical Skills Test, the Cornell Critical Thinking Test, the 
Cognitive Abilities Test, the Watson-Glase * Critical Thinking Appraisal, 
the New Jersey Test of Reasoning Skills, and the SEA Test. 
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Table 2 

Organisation of Understand in, nnA i, WM .,. f ^ Ilf , 1HtMT 

A Triarchir >ro>r» for Tr.i.^p x n r r t Hi _THJLU 

by Robert J, Sternberg 



Students' Test 

Part 1 » Background 

Ch.pt.r 1, S ... Bi.toric.l ..c k! , M . j on vi .«. of I.t.lli,.... aod 

Attempts to Increase It 
Ch.pt.r 2: Th. Tri.rchic Theory of Huwn lot.l lig.oc. 

"• Utmil won, n th . nf ff |n 

Intel l igence 

Chapter 3: Met.components (Executive processes used to plan. 

monitor, and evaluate problem-solving performance) 
Chapter 4: Performance Components (Nonexecutive processes used to 

execute the instructions of metacomponents ) 
Chapter 5: Knovledge-Acquis it ion Components (Nonexecutive processes 
used to learn how eo solve problems) 

?8rt ^ ******** I°divid M .l ? v f „ irffln „ 

Chapter 6: Coping with Novelty 

Chapter 7: Automatizing Information Processing 

Part IV * ^-^ern.l W, r1J_ tf the Individual, Functjons oj gn^ 
Intel li flenep 

Chapter 8: Practical Intelligence 
Pllrt V * I-er«onalitv. Mmt f , , , pd Int^jjj ^g 

Chapter 9: Why Intelligent People Fail (Too Often) 
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Instructor's Manual 

1. Purpose of Chapter 

2. Chapter Outline 

3. Main Ideas 

4. Questions for Class Discussion 

5. Suggested Paper Topics 

6. Supplementary Activities 

7. Suggested Steadings 

8* Suggested Tine Allocation 
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